The Development of Multimedia Evaluation Criteria and a Program of Evaluation for Computer Aided Learning

This paper outlines the evaluation criteria and the approach taken by the Biomedical Multimedia Unit (BMU) to evaluate multimedia technologies being developed in the Faculty of Medicine, Dentistry and Health Sciences at the University of Melbourne. In recent years greater emphasis has been placed on computer based learning in the Faculty. As a result a systematic program of evaluation was developed to monitor the development and implementation of multimedia technologies across the Faculty. Theories of learning and instruction were reviewed and the common features of these models formed the basis for formative evaluation criteria. Evaluation criteria were developed in three domains: instructional and conceptual design, interface and graphic design and user attitudes and affect. In terms of instructional and conceptual design, issues such as the importance of introductory objectives, navigation and orientation, interactivity, sequencing and the need for learning objectives to be consistent with the content of instruction were considered. The evaluation of graphic design was organised by the primary features of the interface such as colour, frames, text, types of media, animations and graphics. These features were evaluated more generally in terms of useability, consistency, clarity, structure (size and spacing), relevancy, usefulness and co-ordination. Features of the interface were also evaluated using criteria of ‘best practice’ which were formed on the basis of previous research in the area (e.g. Boyle, 1997; Hannafin & Peck, 1988; Kearsley, 1985; Oliver & Herrington, 1995; Stemler, 1997). The final domain of formative evaluation assessed users’ attitudes and affect, both more generally and in relation to specific aspects of instructional and graphic design. Once these criteria had been established a program of evaluation which aimed to foster a culture of evaluation in the Faculty was set up. Questionnaires were developed in each of the formative evaluation domains. These questionnaires were administered to those associated with the development of computer aided learning (CAL) modules and CAL module users. The cycle of evaluation was consistent with the CAL module development cycle proposed by the BMU. By adopting a systematic program of evaluation it was envisaged that more educationally effective CAL modules would be produced and developers would become aware of issues which need to be considered when developing CAL modules.